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ABSTRACT 

The purpose, organization, and functions of research 
and evaluation in a large urban school district are discussed in 
light of several lajor issues in educational evaluation. Major issues 
include methods of assuring the credibility of evaluation 
information, the establishment and maintenance of meaningful 
reporting cycles, the recruitment and employment of qualified 
personnel, the release and dissemination of evaluation and research 
data, and the relationship between research, evaluation, planning, 
and development. The role of context, input, process, and product 
evaluation, as well as applied and basic research, in supplying the 
information base for educational planning and decision making is 
highlighted. (Author) 
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The purpose, organization, and functions of 
research and evaluation in a large urban school 
district are discussed in light of several major 
issues in educational evaluation. Major issues 
include methods of assuring the credibility of 
evaluation information, the establishment and 
maintenance of meaningful reporting cycles, the 
recruitment and employment of qualified personnel, 
the release and dissemination of evaluation and 
research data, and the relationship between re- 
search, evaluation, planning, and development. 
The role of context, input, process, and product 
evaluation, as well as applied and basic research, 
in supplying the information base for educational 
planning and decision-making is highlighted. 

- In recent years there has been Increased emphasis on public 
school evaluation. Most of th.^ major public school evaluation 
efforts have been formulated using some modification of several 
conceptual frameworks for evaluation as a guide. The four most often 
encountered frameworks are those attributed to Stuff lebeam, et al., 
(1971), Scriven (1967), Stake (1967), or Provus (1971). A short 
discussion of each of these evaluation structures is offered as 
a background for the forthcoming presentation. 

Probably the most comprehensive evaluation model was developed 
by Stuff lebeam, et al,, (1971), and named the CIPP Model. Evalua- 
tion was defined as the process of delineating, obtaining, and 
providing useful information for judging decision alternatives. 



^Paper presented at the annual meeting of the American Educational 
Research Association, Washington, D.C. , March 31 through April A, 1975. 
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The model identified four major types of evaluation: context 
evaluation to feed planning decisions, input evaluation to feed 
programming decisions, process evaluation to feed implementing 
decisions, and product evaluation to feed recycling decisions. 

Briefly, context evaluation provides a rationale for deter- 
mining educational objectives by defining relevant environment, 
describing desired and actual conditions of the environment, iden- 
tifying unmet needs, and diagnosing problems that prevent needs 
from being met. Input evaluation assesses relevant capabilities 
of responsible agencies and identifies strategies for achieving 
-he objectives determined through context evaluation as well as 
suggesting designs for implementing selected strategies. Once a 
strategy has been selected, process evaluation provides periodic 
feedback to persons concerned with the implementation of plans and 
procedures tc predict or detect faults in procedural design or 
implementation so that interim adjustments may be made if warranted. 
Finally, product evaluation provides interim and final assessment 
of the effects of educational programs. That is, product evalua- 
tion assesses the effects of the strategy selected through input 
evaluation to meet the need identified by context evaluation. Such 
assessment is completed ii. light of process evaluation data. 

Scriven (1967) conceptualized an extremely straightforward and 
widely accepted evaluation framework. It is not nearly so compre- 
hensive as the CIPf Model and is largely concerned with the process- 
product portion cf Stufflebeam's structure. According to Scriven, 



the major goal of evaluation is to make credible judgment rela- 
tive to the merits of educational programs. Within a discussion 
of methods of accomplishing the goal, he introduced the concepts 
of formative and summative evaluation. 

The focus of formative evaluation is upon program improvement • 
Thus, formative evaluation attempts to provide feedback to program 
personnel with the goal of upgrading or improving an educational 
program while it is in the developmental stage. In the CIPP ver- 
nacular, interim product and process data provide formative evalua- 
tion information to program personnel. 

The focus of summative evaluation is upon the determination of 
the ultimate worth of a program or project. This type of evaluation 
should be implemented at that stage in a program's life where it has 
reached some stability. Summative data feed recycling decisions; 
that is, as a result of summative evaluation information, a program 
may be terminated, restructured, continued, or expanded. In the CIPP 
vernacular final product evaluation information, interpreted in 
consideration of context, input, and process data, is used to draw 
summative conclusions about the merits of an educational program and 
feed recycling decisions. 

Stake (1967) suggested that evaluation ought to be concerned with 
three classes of conditions: antecedents, transactions, and outcomes. 
Antecedents are defined as those conditions which exist prior to 
program implementation, i.e., the educational context. Transactions 
are interactions between students, teachers, and materials. Outcomes 
are defined as the intended products of transactions. 
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Three classes of activities were suggested by Stake. The first 
involved providir^ assistance to program staff in generating a clear 
statement of the program or project rationale. The second involved 
the generation of descriptive data. Descriptive data include state- 
ments regarding intended and actual antecedents, transactions, and 
outcomes. Thus a check of the congruence between planned and observed 
antecedents, transactions, and outcomes can be made. Stake also 
suggested an examination of the contingencies within intended (logi- 
cal contingency analysis) and observed antecedents, transactions, and 
outcomes. The contingency analysis within intents is similar to 
CIPP's input evaluation while that within observed data attempts to 
identify cause and effect relationships between antecedents, transac- 
tions, and outcomes. 

The third class of activities involved the generation of judg- 
ments about the worth of educational programs. Stake suggested that 
such judgments be made on the basis of both absolute and relative 
criteria and by a variety of individuals. In other words, programs 
should be assessed both in terms of the degree to which they attain 
absolute, and sometimes arbitrary, gcals and of the degree to which 
they attain those goals relative to other programs with similar goals 
or objectives. 

Provus (1971) suggested that all projects move through design, 
installation, process, and product stages. During each stage the 
evaluator must delineate, in conjunction with project staff, a set 
of standards which can be used as bas .s for comparison with program 
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performance. It is the evaluator*s function to make comparisons 
betweev standards and performance, to identify discrepancies at 
each stage, and to report those discrepancies to project management 
who have the option of terminating the program, proceeding to the 
next stage, or modifying the program in some way. The product of 
the design stage is a set of standards used to judge the effects of 
progran efforts in each of the three succeeding stages. At every 
stage the object of the evaluation is to provide useful data for 
decisions about program improvement or recycling. 

The four generic evaluation frameworks described above are 
supplemented in the evaluation literature by a plethora of articles 
dealing with evaluation methodology, purposes, instrumentation, 
strategies, and variables to be considered in evaluation. In addi- 
tion, the research literature boasts literally thousands of articles 
dealing with research on most areas of the cognitive, affective, and 
psychomotor domains. The more scientific of such studies have been 
called disciplined inquiry by Cronbach and Suppes (1969). According 
to those authors, regardless of whether such inquiry is called eval- 
uation or research (or any number of other terms), a dependence on 
objectivity and evidential test is central. In the words of Cronbach 
and Suppes: 

Disciplined inquiry has a quality that distin- 
guishes it from other sources of opinion and belief. 
The disciplined inquiry is conducted and reported 
in such a way that the argument can be painstakingly 
examined. The report does not depend for Its appeal 
on the eloquence of the writer or on any surface 
plausibility. The argument is not justified by 
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anecdote or casually assembled fragments of evi- 
dence. Scholars in each field have developed 
traditional questions that serve as touchstones 
to separate sound argument from incomplete or 
questionable argument. Among other things^, the 
mathematician asks about axioms, the historian 
about the authenticity of documents, the experi- 
mental scientist about verif lability of observa- 
tions. Whatever the character of a study, if it 
is disciplined the investigator has anticipated 
the traditional questions that are pertinent. 
He institutes controls at each step of information 
collection and reasoning to avoid the sources of 
error to which these questions refer. If the 
errors cannot be eliminated, he takes them into 
account by discussing the margin for error in his 
conclusions. Thus the report of a disciplined in- 
quiry has a texture that displays the raw materials 
entering the argument and the logical processes by 
which they were compressed and rearranged to make 
the conclusion credible...* 

Disciplined inquiry does not necessarily follow 
well-established formal procedures. Some of the 
most excellent inquiry is free-ranging and specula- 
tive in its initial stages, trying what might seem 
to be bizarre combinations of ideas and procedures, 
or restlessly casting about for ideas.... But 
... fundamental to disciplined inquiry is its 
central attitude, which places a premium on objec- 
tivity and evidential test [Cronbach and Suppes, 
1969, pp. 15-16, 18]. 

Research and evaluation in a public school environment must 
attempt to bring disciplined inquiry to bear upon educational deci- 
sions. The model discussed in this paper is an attempt to build 
upon the work of Stuff lebeam and Scriven, as well as that of Bracht 
(1969), Cronbach and Snow (1969), Salomon (1973), and Webster and 
Mendro (1974a, 1974b), in developing a straightforward operational 
model integrating evaluation functions with those of systematic 



applied and basic research. 



The Model 

The research and evaluation model employed in the Dallas 
Independent School District depends heavily on Stuf f lebeam's 
conceptualization of the CIPP Model. ^ The Dallas Model posits 
a comprehensive management-oriented system which highlights the 
interrelationship between evaluation and research while indicat- 
ing the role of each in the process of educational renewal. 
Figure 1.0 presents a flowchart outlining the essential opera- 
tional features of the model. The remainder of this section of 
the paper will be devoted to explicating Figure 1.0. 

A prerequisite to improvement must be a knowledge of exist- 
ing performance levels. Thus, the backbone of any renewal system 
must be a comprehensive context evaluation program. Stufflebeam, 
et al., (1971) define context evaluation as the provision of 
baseline information that delineates the environment of interest, 
describes desired and actual conditions pertaining to the environ- 
ment, identifies unmet needs and unused opportunities, and diag- 
noses the problems that prevent needs from being met and 
opportunities from being used. An adequate context evaluation 
syst^em is founded on a longitudinal data base and provides periodic 
reports on such variables as student dropout, attendance, achieve- 
ment levels, drug usage, demographic and vocational patterns; 
community socioeconomic status and dominant value patterns; and 

^The major exception being a strong dependence in Dallas 
upon the principles of experimental design. 
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teacher academic and demographic characteristics. Thus, a 
context evaluation system provides the basis for formulating 
change objectives by identifying needs and, in some cases, 
outlining practical constraints in identified problem areas. 

Figure 2.0 outlines the basic components of an operational 
context evaluation system. In addition to those investigations 
outlined in Figure 2.0, which are conducted annually, a number 
of specific needs assessments are conducted as they relate to 
specific problem areas. Examples include 3 recent assessment 
of the extent of individualization in District classrooms, a 
recent survey of drug usage among public school children, and 
a current study of the perceptions of patrons, educational 
community, and students regarding the worthiness and effective- 
ness of current and proposed educational practices. 

Figure 2.0. Annual Context Evaluation 

2. 2 Measurement Profiles - a summary of the results 
of the Dictriot's system-wide norm-referenced 
testing program in addition to camunity socio- 
economic data and a profile of the District's 
teaching staff. Results are cumulated by 
school cormu^iity and ai-e presented as they 
relate to national, lax^e city, and local 
noivns. Largely descriptive in nature, these 
profiles are used to inform educators and ^ 
patrons of the relative quality of education 
in the District and to diagnose gross weak- 
nesses in the instructional program. 
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2.2 Criterion'Referenced Testing Profiles - a 
surmary of the results of the District's 
criterion-'referenaed testing program. These 
are used as a supplement to the Measurement 
Profiles and provide estimates of the func- 
tional literacy of school children. 

2.3 Graduate Follow-'Up Stiddies - a series of 
studies on graduates of the District's schools. 
These include comprehensive data on graduate 
employment^ education^ attitudes^ life-status, 
etc., that are used to determine the extent 

to which District programs are meeting student 
needs. The resulting data are then used as a 
guide to curriculum planning. One^-and-' five-- 
year follow-'Ups are conducted. 

2. 4 Dropout Studies - a series of studies designed 
bo provide descriptive data on dropouts, infor- 
mation about variables associated with dropout, 
the interactions among such variables, and trends 
in dropout. Emphasis is on the early identifica- 
tion of potential dropouts so that intei'Vention 
strategies may be implemented. 

2.5 Input Variable Studies - a continuous monitoring 
of the input to District schools. These studies 
provide the cost data for cost-benefit and cost- 
effectiveness analyses, as well as data on the 
s^nhooling environment. 

2.6 Measurement Studies - a series of studies on the 
reliability, validity, and comparability of 
various tests used with District students. These 
studies provide estimates of the degree of faith 
that can be placed in test data. 

2. ? Longitudinal Trend Studies - a series of studies 
investigating achievement, enrollment, and commu- 
nity trends over time. These studies provide an 
accountability function. 

2.8 Teacher and School Effectiveness Indices - a ^ 
system designed to produce student gain statis- 
tics on norm-referenced (fall to fall) and 
criterion-referenced (spring to spring) tests. 
The system identifies teachers and schools that 
are doing better than expected as well as 
teachers and schools that are doing more^ poorly 
than expected, thtts flagging those specific 
situations for additional study. 
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In order to meet fully the Information needs of planning 



decisions, a context evaluation system must Include the capability 
of providing valid projections of the future level of certain im- 
portant variables. Figure 3.0 outlines the general areas of future- 
oriented projection studies that provide crucial Information for most 
major educational decisions. 



Figure 3.0. Projection Studies 

3.1 Student Demographic and Enrollment Study - a 
study designed to project and locate population 
and provide forecasts of future school enroll-- 
ments within specified regions for the purpose 
of providing long^-range planning information 
needed to determine trends and expected demands 
on educational facilities, staff , and programs • 

3.2 Faculty Floio Study - a study designed to pro-- 
Ject the number and cost of teachers reqv.ired 
under a multiplicity of policy and/or environ- 
mental changes. The study will project the 
number and characteristics of teachers who will 
terminate, remain, or need to be hired. Such 
information is useful for teacher cont tct 
evaluation, proposed legislation evaluation, 
staffing projections, and hiring/termination 
analyses. 

3.3 Facilities Study - a study designed to project 
the amount, type, and cost of required space 
areas and to compare projected requirements with 
the existing inventory of space in order to 
determine deficiencies or excesses by individual 
school or demographic area. Such information 
feeds construction and school attendance zoning 
decisions. 

3.4 Financial Study - a study designed to obtain an 
overall financial projection of District needs 
based on input from the preceding studies. 
Feitures include projection of state-^aid fund-- 
ing, debt'-service analyses ind new bond require-- 
ments, revenue and expenditure analyses, and 
tax-rate--demand analyses. 
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These studies encompass many variables and are designed 
to aid decision makers in making intelligent data-based decisions 
about the future. In addition, projection models dealing with 
specific problems, sucn as cafeteria inventor^' ' ^ ordering, 
are designed upon request and receipt of hi^ .ough priority to 
allow funding. 

Once the context evaluation system has identified needs, deci-- 
sion makers must prioritize those needs and focus upon reducing the 
discrepancy between desired and existing conditions by establishing 
goals for those needs that receive highest priority. It is at thi^^ 
point in time that input evaluation information is brought to bear. 
Stuff lebeam, et al., (1971), define input evaluation as the provision 
of information for determining methods of resource utilization for 
accomplishing program goals. In a functioning evaluation system, 
there are four major sources of input information: 

1. previous summative product evaluation information, 

2. basic research information, 

3. applied research information, and 

4. non-rosearch and evaluation information. 

Summative product evaluation information concerns the extent to 
which specific project or program goals are achieved. When product 
evaluation information is available relative to a given program with 
goals similar to those identified in response to context evaluation 
information, that information provides useful input to decision makers 
in determining the probability that the program would reduce the 
identified discrepancy between desired and existing conditions. 
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Basic research information pertains to information about funda- 
mental relationships that affect student learning. Before making a 
decision to implement a given program, decision makers should be 
apprised of the extent to which that program is or is not consistent 
with the principles established by basic research in learning and 
development. 

Applied research information concerns the interaction between 
student characteristics, teacher characteristics, and instructional 
systems. Applied research differs from basic research in that the 
information provided is more closely related to specific decisions 
in an applied educational setting. Decision makers need information 
relative to the types of students (e.g., high anxiety versus low 
anxiety) that function best in given instructional systems implemented 
by teichers with different types of characteristics or traits. Applied 
research must be implemented in the public schools by public school 
research departments. Basic research should probably be implemented 
in the universities in conjunction with the public schools, since many 
local boards of education are probably too impatient for results and 
are unlikely to wait for a long-term, systematic, longitudinal basic 
research program to reach fruition. 

Finally, non-research and evaluation information must enter into 
most educational decisions. Such information as capabilities of staff 
members, costs, political feasibility of program implementation, and 
e:dsting facilities must be taken into account. 

After the collection of relevant input information feeding the 
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preliminary program planning stage, decision makers determine 
whether or not sufficient resources are available to make 
the desired instructional changes. Quite often, adequate 
resources are not available and some compromise is necessary. In 
many cases, the lack of resources is not limited to the realm of 
cost and political feasibility but rather stems from an insufficient 
base of research information. Thus, educators are often in the posi- 
tion of having sufficient material resources but insufficient infor- 
mation resources. 

If sufficient material resources are not available, the system 
may have to exist for some period of time in a state of enlightened 
persistence. Periodic context evaluation will continue to highlight 
the extent of discrepancy between that which is desired and that 
which exists. If the problem results from insufficient information 
resources, programs are often implemented without sufficient support 
data and an information base is built through a series of systematic 
evaluation and applied research studies. 

To cope with the problem of insufficient information resources, 
development centers are established. These centers are charged with 
the responsibility of developing instructional systems to meet the 
needs outlined by context evaluation. Materials and instructional 
systems are developed at the local level only f.f no potentially use- 
ful commercially developed materials are available, since the develop- 
ment of instructional systems is an extremely costly proposition. 
Figure 4.0 outlines the developmental process that theoretically is 
followed in developing instructional systems. Centers serve specific 
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student populations stratified by achievement levels, ethnic back- 

2 

ground, and socioeconomic status. 

The Department of Research, Evaluation, and Information Systems 
provides technical assistance to project management in identifying 
population characteristics, specifying behavioral objectives, and 
conducting task analyses, as well as developing evaluation instru- 
ments and pilot-testing the instructional systems. Once a series 
of instructional systems is validated, the material is ready for 
expanded pilot-testing in schools similar to the setting in which 
the program was developed. 

If sufficient material and information resources are available, 
or if sufficient material and minimal information resources are 
available, the extended program planning nhase is entered. This is 
the phase that is entered as a result of tlie information gleaned from 
the input evaluation. The 3valuator*s role in the earlier phase of 
program planning involved making all relevant, available input infor- 
mation available to program planners. Once it is decided to take a 
particular course in remediating a demonstrated need, the evaluator 
must ensure that stated program objectives are measurable. Technical 
planners, also on the staff of the Department of Research, Evaluation, 
and Information Systems provide technical assistance in planning pro- 
ject implementation. 

Out of the program planning sessions, the evaluator develops a 

detailed program evaluation design specifying the criteria by which 
the program will be judged. The development of this evaluation desigi. 

^At this time, the Dallas Independent School District has two 
developmental centers. 
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must necessarily involve continuous interaction between program per- 
sonnel and the evaluator in order ultimately to produce maximally 
effective information. Obviously, the evaluator must be independent 
of program management to ensure the optimum objectivity of evaluation 
results. 

Once the program implementation phase is entered, the evaluator *s 

i 

role becomes one of providing continuous formative evaluation reports 
relative to program implementation. These reports fall primarily into 
two categories, process evaluation and interim product evaluation. 
Process evaluation has three major objectives: (1) the detection or 
prediction of defects in procedural design or its Implementation 
during program implementation stages; (2) the provision of information 
for programmed decisions; and (3) the maintenance of a record of the 
implementation procedure as it occurs (Stuff lebeam, et al., 1971). 
Thus, process evaluation information keeps program management in- 
formed of the extent to which program implementation conforms to 
specifications and from an evaluation standpoint, guards against 
the e%^aluation of a fictitious event. 

Interim product evaluation provides periodic feedb^ick to program 
management relative to the attainment of specific subobjectives during 
the implementation phase. Thus, process and interim product evalua- 
tion reports inform program management as to implementation and goal 
attainment levels while program adjustments are still feasible. 

Upon completion of a given cycle of program implementation, a 
summative product evaluation report is prepared. This report gener- ^ 
ally addresses three areas of concern: (1) the extent to which program 
objectives were achieved relative to some specific set of criteria; 
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(2) the extent to which system objectives were achieved relative to 
alternative instructional strategies; and (3) the cost-effectiveness 
of the program relative to alternative instructional strategies. It 
should be obvious that information relative to these three areas of 
concern must be interpreted in light of process and interim product 
evaluation information. Without information about program implemen- 
tation, product evaluation information is of little use. 

Figure 5.0 outlines the necessary steps in project evaluation* 
The reader should bear in mind that project evaluation depends on 
strong experimental design and commences at that point that the need 
has been identified, resources have been allocated, and the program has 
been tentatively planned. The evaluator does not have to aid in pro- 
gram planning to evaluate the program. 

Figure 5.0. Necessary Steps In Project Evaluation 

5. 1 Program Objectives Determination 

5.1.1 Meet with decision makers and program 
managers to determine the program 
objectives. 

5.1.2 Refine objectives through thorough 
analysis^ review of literature^ ques- 
tioning decision makers^ analysis of 
input data^ etc. 

5.2 Information Regarding Program Decisions 

5.2.1 Using the objectives, meet with decision 
makers^ etc. to generate a list of the 
critical decisions to be made concerning 
the objectives and the program. 

5.2.2 Determine the types of information nec- 
essary to make the various decisions. 

5.2.3 Estimate the critical decisions and plan 
the information sources so critical 
decisions re ive the most information. 
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5.3 Definition of Measurable Objeotives md Related 
Decisions 

5.3.1 Work with project personnel to mold 
objectives so that they may be measured. 

5.5.2 Operationalize basis for decisionmaking 
to relate to measured achievement of 
objectives. 

5.4 Plan of EvaliAation Dissemination 

5.4.1 Identify the various audiences of the 

evaluation report and estimate the level 
of sophistication of the intended 
audience. 

5.5 Identification of Measuring Instruments 

5.5.1 Review objectives and decisions and 
evaluate existing instruments to deter- 
mine those which can be employed in the 
evaluation. 

5.5.2 Determine areas where no satisfactory 
instruments are available and develop 
complete specifications of instruments 
that are to be constructed. 

5.6 Instrument Development and Testing 

5.6.1 Develop needed instruments. 

5.6.2 Test new instruments^ if necessary ^ on a 
sample of subjects. 

5.6.3 Refine new instruments on the basis of 
these tests. 

5.6.4 Test administration of any unconventional 
instruments or observation procedures. 

5. 7 Information Collection Scheduling 

5.7.1 Specify sampling procedures to be employed. 

5.7.2 Determine the schedule of observations and 
the instruments to be aSninistered at each 
observation point. 

5.7.3 Schedule the personnel needed to administer 
instruments. 

5.8 Organization of Data Analysis 

5.8.1 Determine various formats of data including 
card and tape format specifications at 
vaHous stages of collection and analysis. 
Specify processing necessary to put data 
into correct format at each stage of 
analysis. 
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5.5.2 Plan nonstatistical analysis of data and 
resources necessary to perform analysis. 

5.8.3 Plan statistical analysis of data and 
programs necessary to analyze data. 

5.8. 4 Determine which programs are already 
written and are ready to use^ which pro- 
grams are written but need modifications 
to handle data in their intended formats y 
and which programs need to he written 
with specifications of these programs. 

5.9 Formal Evaluation/Research Design 

5.9.1 Prepare design including specification of 



5.9.1.5 Sampling procedures 
5.9.1*6 Data analyses schedules 
5.9.1.7 Final reporting schedules 

5.9.2 Typey prints and collate design. 

5.9.3 Disseminate formal design^ 

5.10 Computer Program Development 

5.10.1 Develop necessary programs for analysis 

5.10.2 Make necessary modification of existing 
programs. 

5.10.3 Run all programs to he used on sample 
data in the proper medium and format. 
Construct sample data to simulate ^ 
problems in actual data (mispunching ^ 



6.11 Process Evaluation 

5.11.1 Collect or supervise and coordinate 
collection of process evaluation 
information. 

5.11.2 Prepare process evaluation information 
for analysis. 

5.12 Produ-it Evaluation 

5.12.1 Collect or supervise and coordinate 
the collection of product evaluation 
information. 

5.12.2 Prepar*e product evaluation information 
for analysis. 



5.9.1*1 
5.9.1.2 
5.9.1.3 
5.9.1.4 



Objectives 

Instrumentation 

Analysis methodology 

Data collection and reporting 

schedules 



missing data^ etc.). 
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5. IS Interim Data Analysis 

5.13.1 Organize interim data. 

6.13.2 Perform analysis of interim data. 

S. 14 Fomative Evaluation Reports 

5.14 A Prepare formative evaluation reports. 

5.14.2 Type, print, and collate formative 
evaluation reports. 

5.14.3 Disseminate fomative evaluation 
reports to project management and 
staff. 

5. 15 Swwmtive Data Analysis 

5.15.1 Organize swmative data. 

5.15.2 Perform analysis of summative data. 

5.16 Summative Evaluation/Research Reports 

5.16.1 Prepare the various summative evalua-- 
tion/research reports for each audience 
including objectives, findings, and 
recommendations expressed in an appro- 
priate manner for the intended audience. 
This preparation includes the abstract 
of the report. 

5.16.2 Have report carefully proofread and 
corrected. 

5.16.3 Type, print, and collate the sumative 
evaluation reports. 

5.16.4 Disseminate the summative evaluation/ 
research reports to project personnel. 
District management, and the Board of 
Education. 

5.17 Interpretation of Reports 

5.17.1 Meet with project personnel to inter-- 
pret reports. 

5.17.2 Meet with District management and the 
Board of Education to aid in report 
interpretation. 

5. 18 Further Report Dissemination 

5. 18. 1 Disseminate summative evaluation/research 
reports to all affected District adminis^ 
trators and to interested professional 
staff. 

5.18.2 Prepare and disseminate a book of evalua-- 
tion and research abstracts to all pro-- 
fessional staff. 
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5.19 Report Feedback 

S.19.1 Meet with decision makers to obtain 
feedback regarding the report h)ith 
the purpose of improving reporting 
activities. 

Many evaluation systems stop at the provision of product eval- 
uation Inforffiatlon. These data generally bear upon the performance 
of different groups of students under va^lng treatment configura- 
tions. Unfortunately, as a result, most product evaluation reports 
have generally focused on the search for single best treatments for 
all learners, i.e., main effects. In order to provide needed infor- 
mation for educational decision makers, applied research studies 
involving the systematic investigation of aptitude-treatment and 
trait-trait Interactions must be undertaken. Such studies would be 
expected to provide important information relative to replicable 
relationships between student, teacher, and program characteristics 
(Bracht, 1969; Cronbach and Snow, 1969; Salomon, 1972; Webster and 
Mendro, 1974b). 

The basic assumption of aptitude-treatment interaction research 
is that learners possess characteristics or traits that Interact 
positively or negatively with specific treatments or program charac- 
teristics* Messick (1970) outlined some cognitive style dimensions 
which represent a person's typical modes of perceiving, remembering, 
thinking, and problem solving, and, as such, would provide excellent 
variables for aptitude- treatment interaction studies. Figure 6.0 
presents Messick*? exariples of these dimensions. 
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Pigur^c 6.0. Examples of Cognitive Styles (Messick^ 1970) 

6.1 Field independence versus field dependence - 
"an analytical^ in contrast to a global^ my 
of perceiving (which) entails a tendency to 
experience items as discrete from their hack- 
grounds and reflects ability to overcome the 
influence of an embedding context" (Witkin 
et al.y 1962). 

6. 2 Scanning - a dimension of individual differences 
in the extensiveness and intensity of attention 

' deployment^ leading to individual variations in 
viviStess of experience and the span of aware-- 
ness (Holzmann, 1966; Schlesingev^ 1954; Gardner 
and Longy 1962). 

6. 3 Breadth of categorizing - consistent preferences 
for broad inclusiveness ^ as opposed to narrow 
exciusivenessy in establishing the acceptable 
range for specified categories (Pettigrew^ 1958; 
Bruner and Tajfel^ 1961; Kogan and Wallach^ 1964). 

6.4 Conceptualizing styles - individual differences ^ 
in the tendency to categorize perceived similari-- 
ties and differences among stimuli in terms^ of 
many differentiated concepts ^ which is a dimension 
called conceptual differentiation (Gardner and 
Schoeny 1962; Messick and Kogan^ 1963) ^ as well 

as consistencies in the utilization of particular 
conceptualizing approaches as bases for forming 
concepts - svch as the routine use in concept 
formation of thematic or functional relations 
among stimuli as opposed to the analysis of 
descriptive attributes or the inference of class 
membership (Kaqan^ Moss^ and Sigel^ 1960; Kagan 
et al.y 1963). 

6. 5 Cognitive complexity versus simplicity - indi- 
vidual differences in the tendency to construe 
the worldy and particularly the world of social 
behavior^ in a multidimensional and discriminat-- 
ing way (Kelly, 1955; Bieri, 1961; Bieri et al., 
1966; Scott, 1963; Harvey, Hunt, and Schroder, 
1961). 

6.6 Reflectiveness versus impulsivity - individual 
consistencies in the speed with which Hypotheses 
are selected and information processed, with im^ 
pulsive subjects tending to offer the first ^ 
answer that occurs to them, even though it is 
frequently incorrect, and reflectije subjects 
tending to ponder various possibilities be Core 
deciding (Kagan, et al., 1964; Kagan, 1965). 
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6. 7 Leveling versus sharpening - reliable individual 
variations in assimilation in memory. ^Subjects 
at the leveling extreme tend to blur similar 
memories and to merge perceived objects or 
events with similar but not identical events 
recalled from previous experience. Sharpeners^ 
at the other extreme, are less prone to confuse 
similar objects and, by contrast,^ may even 
judge the present to be less similar to the past 
than is actually the case (Holzman, 1954; 
Holman and Klein, 1954; Gardner et al.j 
1959). 

6. 8 Constricted versus f lexible control - individual 
differences in susceptibility to distraction 
and cognitive interference ^ Klein, 1954; Gardner 
et al.y 1959). 

6. 9 Tolerance for incongruous or unrealistic ex- ^ 

perienoes - a dimension of differential willing- 
ness to accept perceptions at variance .dth 
conventional experience (Klein et al. , 1962). 

In addition, there are many affective variables that warrant 
investigation when attempting to validate replicable teacher-student 
(trait-trait) interactions. The basic assumption of trait-trait 
interaction research is that teachers possess characteristics or 
traits that, independent of program, interact positively or nega- 
tively with specific characteristics or traits posseissed by 
learners. Such interactions may involve variables such as arith- 
metic reasoning, language usage, vocabulary, abstract reasoning, 
mechanical reasoning, creativity, anxiety, affiliation, aggressive- 
ness, compulsivity, dogmatism, paranoia, and status variables such 
as sex, age, or ethnicity (Webster and Mendro, 1974b). 

Once context, input, process, and product evaluation informa- 
tion, as well as applied research data are available, non-research 
and evaluation information once more is brought to bear upon the 



decision-making process. It would be naive to expect educational 
decisions to be made purely on the basis of research and evaluation 
data. Once again, information such as the absolute program costs, 
capabilities of program staff members, political feasibility of 
program implementation, and existing facilities and resources must 
be considered by decision makers in rendering a final decision. 

In determining the fate of a given program, four primary choices 
are available to decision makers. First, they can choose to continue 
the program in its current setting. If this alternative is chosen, 
the summative product evaluation report and the applied research 
data become the context evaluation information for the next implemen- 
tation phase, and program implementation commences. This alternative 
generally occurs when decisions are to be made on the basis of longi- 
tudinal studies, i.e., where it is expected that results will not 
be in evidance after a relatively short rlmplementation period. 

A second alternative is to discontinue the program. This is 
usually done after product evaluation studies demonstrate the failure 
of the progr'^.m to meet its objectives or in those cases where the 
program is simply not cost-effective. (Failure to meet objectives 
is often a necessary, but not sufficient, condition for program dis- 
continuation.) Once a program is discontinued, the system returns to 
context evaluation and once again applies the needs assessment and 
orientation phases. 

If the product evaluation and applied research information are 
favorable, and it is practically and politically feasible, the program 
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may be expanded to serve additional students. Prior to making the 
decision to expand the program, additional context evaluation in- 
formation must be examined to determine if similar needs exist in 
other settings. If such needs are demonstrated, then the program 
may be expanded to other settings and the program planning stage 
entered to extend the program implementation. Other settings eli- 
gible for program expansion may include entire schools or specific 
subpopulations (e.g., highly motivated students) as indicated by 
applied research data. If such needs do not exist, the program is 
continued with the original target * population or a reduced tai;get 
population based on the results of the applied research studies. 
The extent of continued evaluation to be implemented under either 
the expansion or continuation alternative is determined by decision 
makers under advisament from evaluation personnel. 

A fourth alternative involves program revision. Much program 
revision should be accomplished on the basis of process and interim 
product evaluation reports. Often, however, summative product 
evaluation and applied research reports reveal weaknesses in portions 
of programs that would otherwise appear to be functional. In this 
instance, the summative product evaluation and applied research re- 
ports become the context evaluation information for the next program- 
planning cycle. 

Figure 1.0 outlined a model for an operational research and 
evaluation system. However, data, no matter how comprehensive, are 
of limited use if not disseminated to decision makers. Figure 7.0 
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outlines the information requirements of decision makers at different 
management levels. 



Figure 7. 0. Information Kequivments Of Various Manage- 
ment Levels That Should Be Met Through 
Systematic Research And Evaluation Programs 
(Webster and Mer.dro, 1974). 



Management Levels 

Board of Education 
(Public) 



Upper Management 



Project Management 



Information Requ-^'rements 

Product evaluation Infomation 
(generally summative) on the 
effects of District projects 
and programs 

Context evaluation information 
on the general state of the edu- 
cational environment 

Product evaluation information 
(generally swmative) on the 
effects of District projects 
and programs 

Context evaluation on the general 
state of the educational environ- 
ment 

Basic research informatior (mostly 
literature reviews) on selected 
fundamental relationships in 
learning 

Input evaluation information rela- 
tive to alternative strategies for 
meeting specific needs 

Applied research information on 
the interactions between student 
characteristics 3 teacher charac- 
teristicsy and program success in 
all District programs 

Product evaluation information of 
a formative and sumnative nature 
on the effects of specific projects 

Context evaluation infomation on 
the baseline state of the project 
environment 



ERIC 



27 



Process evaluation information on 

a continuing basis relative to 

the extent of program implementation 

Applied research information on the 
interaction between student charac- 
teristics, teacher characteristics , 
and program success 

Basic research information on 
selected fundamental relationships 
in learning 

Building Management Product evaluation information 

(generally summative) on the effects 
of District projects and programs 

Context evaluation information on 
the general state of resident 
teachers and students and the 
sponsoring community 

Input evaluation information rela- 
tive to alternative strategiea for 
Tueeting specific needs 

Formative and sumativa product 
evaluation and applied research 
information on specific projects 
implemented in the building 

Teachers Product evaluation information of 

a formative and swmative nature 
on the effects of specific programs 
in which the teacher is involved 

Context evaluation information rela- 
tive to student entry level 

Process evaluation information on 
degree of program implementation for 
specific programs in which the 
teacher is involved 

Applied research information rela-- 
tive to the types of instructional 
strategies that are most effective 
with different learner characterise 
tics and specific validated instru-- 
ments and instructions for their 
use in student diagnosis 

Parents/Students Feedback on individual student 

performance 



In order for research and evaluation information to be maximally 
effective, structured systems must be established to inject the empiri- 
cal data into the decision-making process. This must be accomplished 
through the establishment at the local building and departmental level 
of zero-based budgetary planning models which systematically include 
research and evaluation data as part of the information needed 
for planning. At a higher level* there must be a method of supplying 
research and evaluation ds-ta to the Board of Education and public. 
In Dallas > the accountability function is performed through the 
Educational Program and Evaluation Committee of the Board of Education, 
.Figure 8.0 outlines the functions of that Committee. 

Figure 8.0. Functions Of The Educational Program and 
Evaluation Comnittee Of The Board Of 
Education 

The primary missions of research and evaluation 
in the public schools are to provide acctdrate infor- 
mation in usable form to decision makers and to 
fulfill an accountability function. The Board of 
Education in its decisions-making role relative to 
District goals ^ priorities^ and policies will be 
a primary user of p^valuation information. Addi- 
tionally^ the Board will have the responsibility 
for the definition of guidelines relative to its 
role in the utilization of program evaluation in- 
formation. The following policies address this 
responsibi li ty : 

8.1 All educational program evaluation reports 
shall be submitted to the Board Educational 
Program and Evaluation Committee for study 
and recormendations to the Board of Education. 

8.2 The Board of Education evaluates continuously 
and reports regularly to the public all activi- 
ties of the Dallas Independent School District. 

8.3 Only the Board of Education Educational Pro- 
gram and Evaluation Committee may release 
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cdUi^'ational pvoijvam evaluation vcporta to 
individuals or agencies outside the Dallas 
Independent School District, 

In order to define a systematic procedure by which 
evaluation reports may be brought to the attention of 
the Boards the following steps have been specified: 

8.4 All educational program evaluation reports of 
the Dallas Independent School District will be 
forwarded to the executive officer in charge of 
the Development Division prior to their release 
to appropriate decision makers. 

8. 5 At the discretion of the executive officer in charge 
of the Development Division^ the evaluation reports 
will be submitted to appropriate staff members for 
their suggestions concerning possible revisions. 

8.6 The executive officer in charge of the Development 
Division will submit all educational prognvn evalua-^ 
tion reports to the Educational Pvogrcffn and Evaluation 
Committee for their information and disposition. 

As the Board Cormittee concerned with evaluation 
andy subsequently y the utilization of evaluation re- 
ports y functions of the Educational Program and 
Evaluation Committee of the Board of Education are ^ 
relevant. These aforementioned functions are speci- 
fied below: 

8.7 Continuously assess and report to the Board the 
School District's needs for program evaluation 
information. 

8.7.2 Monitor^ review^ and report to the Board 
program evaluation information as this 
information becomes available. 

8. 7. 2 Identify and report to the Board unmet 
program evaluation information needs. 

8.8 Identify alternatives and recommend priorities 
to the Board for providing needed evaluation 
information. 

8.8.2 Assess the consequences of alternative 
approaches to providing needed evalua- 
• tion information. 

8. 8. 2 Report to the Board recommended priori- 
ties for providing needed program 
evaluation information. 
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5. 9 Report to the Board veoormended priorities for 
obtaining and utilizing fiends and other re- 
sources in educational program evaluation. 

8. 9. 1 Receive and review communications frorn^ 
outside institutions that provide funds 
and other resources for educational 
program evaluation. 

8.9.2 Receive and review management recommen-* 
dations regarding utilization of edu- 
cational program evaluation resources. 

8.10 Evaluate and report to the Board the School 
District's educational program evaluation 
effort. 

8.10.1 Receive and review reports of all in- 
house and external educational evalua- 
tion, educational auditj and cost- 
effectiveness sti4dies and reports* 

8.10.2 Report to the Board the Comittee's 
evaluation of bhe School District's 
educational program evaluation efforts. 

Probably timeliness is the single most important factor in deter- 
mining whether or not an evaluation report is useful. An evaluation 
report that is submitted after 

Since adequate time must be available to collect, analyze, and re- 
port data and an educational program must be given sufficient time 
to demonstrate an effect, the Dallas Independent School District has 
instituted a fifteen-month planning cycle. That is, an evaluation 
conducted during 1974-75 does not materially affect 1975-76 planning 
but, rather, feeds into the 1976-77 budget request that is prepared 
by December, 1975. This is true only of project evaluation, since 
all other types o^ evaluation can be legitimately completed early 
enough in the school year to allow their immediate use in decision- 
making. Since most project evaluations in Dallas are th^ee-year 
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cumulative longitudinal designs, this planning approach often results 
m decisions being made on the basis of two-and-one-half years of data- 
This system allows for the production of quality evaluation and re- 
search reports that significantly affect educational decisions- 
Organization 

Figure 9-0 outlines the basic organization of a department de- 
signed to produce the types of information specified in Figure 7,0. 
The organizational chart is by function. Obviously, the size of any 
given branch would depend upon the breadth of information requirements 
of the Board of Education, Administration, and teaching staff that it 
services - 

Figure ^.0. The Organization Of A Functional Research And 
Evaluation Unit 



Administration. 



Administrative 
Staff 



Testing 



Developmental Project 
Evaluation 



System-Wide 
Testing 



Instrument 
Development 



Special Project 
Evaluation 



Operations 
Planning 



System-Wide Evaluation 



Administrative Research 



Institutional 
Research 



Long-Range 
Planning 



Applied 
Research 



Computer 
Applications, 



Ad Hoc 
Information 
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Administration 

The Department of Research, Evaluation, and Information Systems 
Is administered by technicians with management skills. Too often 
research organizations In applied settings are administered by non- 
technicians who at times underestimate the scope of requested services, 
thus overcommitting resources and failing to meet deadlines, and who 
must rely on subordinates to hire competent personnel. Since it Is 
very difficult for nontechnical aoministrators to hire technical 
staff, and since the quality of an evaluation effort is almost 
completely dependent upon the ability and committment of the evalua- 
tion staff, it behooves chief administrators in applied settings to 
hire research managers trained in the basic skills of systematic in- 
quiry to head research departments. 

The administration branch has major responsibility for the 
management of all research and evaluation programs being Implemented 
in the District. Members of this branch participate in the design, 
interpretation, report-writing, and dissemination phases of all major 
evaluation efforts and consult with District decision makers and the 
Board of Education. They procure and allocate resources, assign 
priorities, and make decisions concerning the validity of research 
and evaluation designs, testing, statistical applications to ques- 
tions in the behavioral science domain, process and product evaluation 
methodology, measurement strategies, and data interpretation and 
analysis. The administrative group must broker the politics of the 
system in such a manner that the independence of project and program 
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eval'jators is assured and objective data from systematic, controlled 
research and evaluation studies are injected into the decision-making 



process. 

The administrative team consists of the department head, an ad- 
ministrative assistant to the department head, a senior technical 
assistant to the department head, and four heads of the four branches 
of the department (Testing, System-Wide Evaluation, Developmental 
Project Evaluation, and Administrative Research). Figure 10.0 out- 
lines the optimal competencies for each member of the administrative 
team. These competencies are listed in their entirety in the Appendix. 
Close scrutiny of the competencies will shed light on the functions of 
each of the positions, since only those competencies that contribute to 
optimal functioning are listed. Actual job descriptions list minimal 
and optimal competencies. Minimal competencies are a subset of the 
optimal competencies. 

Figia>e 10.0. Optimal Competencies For Research Manage- 
ment Personnel* 

Department Head U 2, 3, 6, 16, 19, 22, 23, 24 26 27 

^ 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 

39, 40, 45, 46, 49, 50, 51, 52, 55, 57, 

58, 59, 62, 63, 64, 65, 67, 68, 69, 70, 

76, 80, 85, 91, 98, 99, 100, 101, 102, 103, 

104, 105, 106, 107, 108, 109, 110, 111, 

113, 114, 116, 117, 123, 124, 125, 126, 
127, 128, 129, 130, 132, 133, 134, 135, 
126, 137, 138, 141, 142, 143, 144, 145, 
146, 152, 154, 155, 156, 158, 159, 160, 
161, 162, 163, 164, 165, 166, 169, 170, 
171, 172, 173, 174, 175. 

Senior Technical 1, 2, 3, 6, 7, 10, 16, 17, 18 19 24 
Assistant 29, 30, 31, 32, 33, 34, 35, 36 45 46 

49, 50, 51, 52 57, 55, 59, 60, 63, 64, 
65, 67, 70, 71, 73, 76, 77, 78, 79, 80, 
81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 
91, 92, 93, 94, 95, 98, 100, 102, 103, 104, 

105, 106, 107, 108, 109, 110, 111, 113, 

114, 116, 117, 122, 123. 

*fha competencies Usted are those r enquired -to functio:: 
&pt'umally in the positions specified. 
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Senior Technical 224, 226, 227, 232, 232, 233, 234, 235, 
Assistant (cant.) 236, 237, 238, 239, 240, 242, 242, 243, 
244, 245, 246, 247, 248, 249, 252, 253, 
254, 255, 256, 257, 258, 259, 260, 262, 
2S2, 265, 266. 

Administrative 2, 2, 3, 6, 9, 12, 23, 26, 29, 20, 22, 
Assistant 22, 24, 27, 33, 45, 63, 65, 67, 68, 69, 

76, 92, 98, 99, 200, 203, 204, 205, 206, 
207, 125, 228, 229, 230, 232, 234, 262, 
263, 264, 265, 267, 268, 172, 273, 274, 
275. 

Hrector-- 2, 2, 3, 6, 26, 19, 22, 23, 24, 27, 30, 

Testing 32, 33, 34, Z5, 36, '67, 38, 39, 40, 45, 

46, 49, 50, 51, 52, 55, 56, 57, 58, 59, 
60, 62, 62, 63, 64, f-l, 66, 67, 68, 69, 
70, 72, 72, 73, 76, 80, 85, 92, 98, 200, 
202, 203, 204^ 105, 206, 207, 220, 223, 
226, 227, 127, 128, 130, 132, 134, 135, 
136, 137, 138, 141, 145, 146, 154, 156, 
158, 159, 160, 161, 162, 163, 164, 165, 
166, 169, 170, 171, 173, 174. 

Director - 1, 2, 3, 6, 7, 10, 16, 17, 18, 19, 22, 23, 

System-Wide 24, 26, 27, 30, 31, 32, 33, 35, 36, 37, 38, 

Evaluation 39, 40, 45, 46, 49, 50, 51, 52, 53, 54, 55, 

57, 58, 59, 61, 63, 64, 65, 67, 68, 69, 76, 

77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 
88, 89, 90, 91, 93, 94, 95, 98, 100, 101, 102, 
103, 104, 105, 106, 107, 108, 109, 110, 111, 
113, 116, 117, 122, 123, 124, 125, 126, 127, 128, 
130, 132, 133, 134, 135, 136, 137, 138, 139, 140 
141, 144, 145, 146, 147, 148, 14y, 152, 153, 154, 
155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 
165, 166, 1.69, 170, 171, 173, 174. 

director - 1, 2, 3, 6, 10, 16, 19, 22, 23, 24, 25, 26, 27, 

Developmental 28, 29, 32, 33, 34, 35, 36, 37, 38, 39, 40, 45, 
-Project 46, 49, 50, 51, 52, 53, 54, 55, 57, 58, 59, 60, 

Evaluation 63, 64, 65, 67, 68, 69, 70, 76, 77, 78, 80, 85, 

91, 98, 100, 101, 102, 103, 104, 105, 106, 107, 
108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 
125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 
135, 136, 137, 138, 141, 142, 143, 145, 246, 151, 
152, 154, 155, 156, 158, 159, 160, 161, 162, 163, 
164, 165, 166, 169, 170, 271, 173, 174. 
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Diveotor - 1, 2, 3, 6, 7, 10, 16, 17, 18, 19, 23, 

Administrative 24, 26, 27, 32, 33, 35, 36, 37, 38, 39, 
Reeearoh 40, 45, 46, 49, 50, 51, 52, 55, 56, 57, 

63, 64, 65, 67, 76, 77, 78, 79, 80, 81, 
82, 83^ 84, 85, 86, 87, 88, 89, 90, 91, 
92, 94, 96, 98, 100, 101, 102, 103, 104, 
105, 106, 107, 108, 109, 110, 113, 116, 
123, 124, 126, 127, 128, 130, 131, 132, 
134, 135, 136, 137, 138, 139, 140, 141, 
145, 146, 152, 155, 156, 158, 160, 161, 
162, 163, 164, 165, 166, 169, 170, 171, 
173, 17 

Assistants - 2, 3, 5, 3, 7, 9, W, 14, 16, 19, 20, 21, 

Administration 24, 41, 42, 43, 44, 45, 46, 55, 56, 57, 63, 
65, 67, 74, 75, 76, 77, 85, 98, 99, 100, 
101, 103, 104, 105, 106, 107, 109, 113, 
116, 123, 124, 128, 129, 130, 132, 134, 
165. 

Support - 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 

Administration 16, 19, 56, 74, 75. 



In addition to those positions specified above, a number of 
specialized administrative personnel must be retained by a large re- 
search and evaluation department. These are not decision makers but 
rather support personnel. Two important examples include an editor 
and a production controller (in charge of producing the reports). 

Perusal of Figure 10.0 suggests that the senior technical assis- 
tant is a technician while the administrative assistant actually helps 
in the daily routine management of the department. The senior tech- 
nical assistant functions as the quality control officer as well as 
aids the department head in many daily tasks. He is responsible for 
assuring the accuracy and objectivity of all evaluation and research 
reports. Reports are reviewed, data collection and analysis proce- 
dures verified, and results certified. External auditors have 
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occasionally been employed for this purpose, however, due to their 
general inability to become sufficiently familiar with applied 
evaluation problems and methodology in specific settings, have 
been of limited utility. The senior technical assistant is charged 
with the responsibility of keeping abreast with the latest trends 
and developments in evaluation, research, measurement, and statisti- 
cal methodology, and is drawn from the ranks of the best evaluators 
in the department. 

Testing 

The testing branch must have the capability to plf>n ano imple- 
ment a large-scale testing program as well as to develop and validate 
criterion-referenced tests and other specialized instruments. It 
functions as a service branch to '^ther branches of the department in 
developing instruments for use in applied research and program eval- 
uation applications. It must maintain an active file on available 
instruments so that much duplication in instrument development can 
be avoided. 

The System-Wide Testing office has major responsibility for the 
development and implementation of policies and procedures concerning 
the collection, quality control, availability, and utilization of 
group system-wide testing data. It formulates new strategies for 
measuring and reporting student aptitude and achievement, keeps 
abreast of the literature in measurement and testing, and communi- 
cates with the principals and building test coordinators relative 
to testing procedures and interpretation. 
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The Instrument Development and Validation office has major 
responsibility for the development and validation, in conjunction 
with subject-area consultants and research and evaluation user 
personnel, of criterion-referenced tests and affective instruments. 
An active instrument file is maintained by this office. Figure 11.0 
outlines the optimal competencies of personnel in the various test- 
ing offices. 

Figure 1.1.0. Optimal Competencies Foy Testing Personnel* 

Senior Staff - 1, 2, Z, 6, 7, 10, 16, 19, 22, 23, 24, 26, 
System-Wide 27, 32, 33, 35, 36, 45, 46, 49, 50, 51, 52, 

Testing 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 

64, 65,, 66, 67. 68, 69, 70, 71, 72, 73, 76, 77, 
80, 85, 91, 98, 100, 102, 103, 104, 105, 106, 
107, 110, 113, 116, 117, 127, 132, 134, 135, 
136, 137, 138, 141, 145, 146, 156, 158, 160, 
161, 162, 165, 166. 

Assistants - 2, 3, 5, 6, 7, 9, 10, 13, 14, 16, 19, 22, 23, 
System-Wide 24, 41, 42, 43, 44, 45, 46, 55, 56, 57, 58, 

Testing 59, 63, 65, 66, 67, 68, 69, 70, 72, 74, 75, 76, 

77, 85, 98, 100, 103, 104, 105, 106, 107, 113, 

116, 134, 165. 

Senior Staff - 1, 2, 3, 6, 7, 10, 16, 19, 22, 23, 24, 26, 27, 
Instrument 32, 33, 35, 36, 41, 42, 43, 44, 45, 46, 47, 

Development 48, 49, 50, 51, 52, 53, 54, 55, 5P, 57, 58, 

59, 60, 61, 63, 64, 65, 66, 67, 68, 69, 70, 71, 
72, 73, 76, 77, 78, 85, 91, 98, 100, 102, 103, 
104, 105, 106, 107, 110, 113, 116, 117, 119, 
132, 134, 135, 136, 137, 138, 141, 145, 146, 
156, 158, 160, 161, 162, 165, 166. 

Assistants - 2, 3, 5, 6, 7 9, 10, 13, 14, 16, 19, 23, 24, 
Instrument 41, 42, 43, 44, 45, 46, 47, 48, 55, 55, 57, 

Development 58, 59, 60, 63, 65, 66, 67, 68, 69, 70, 72, 74, 

75, 76, 77, 85, 98, 100, 103, 104, 105, 1C6, 

107, 113, 116, 134, 165. 

Support - 2, 3, 4, 5, 6, 7, 8, 9, 12, 23, 14, 15, 16, 

Testing 19, 41, 56, 74, .'5. 



*The aompetenaies listed are those required to function optimally in 
the positions specified. 
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Management: In situations where individuals are charged 

with the management of subunitSf competencies 
163^ 164 y and 170 also apply. 

System-Wide Evaluation 

The System-Wide Evaluation branch is responsible for all longi- 
tudinal and cross-sectional system-wide planning, research, and 
evaluation programs. It operates through three offices: Institutional 
Research, Long-Range Planning, and Applied Research. 

The Institutional Research office is responsible for the design 
and implementation of studies that utilize the District's various 
longitudinal data bases. The various types of studies conducted by 
this office were outlined in Figure 2,0. Such studies require a sophis 
ticated knowledge of data processing and multivariate statistics as 
well as research techniques developed through disciplines other than 
psychology and education (e.g., path analysis). 

The Long-Range Planning office is charged with the responsibility 
of developing computerized simulation models designed to improve the 
information base from which decisions are made. It is the purpose of 
tliese models to predict the outcome of policy and/or environmental 
changes prior to the implementation of these changes and thus bring 
data to bear upon the operationalization of future-o^^ented decisions. 
General areas of focus were outlined in Figure 3.0. 

The Office of Applied Research is responsible for the systematic 
investigation of individual differences in learning. It develops and 
utilizes strategies for investigating Interactions among student, 
teacher, and treatment configurations that produce optimal student 
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learning. This office also coordinates university research within 
the District and issues requests for proposals to investigate spe- 
cific problem areas. The key to the success of this type of en- 
deavor would seem to be a system^atic longitudinal program of mutually 
supporting studies designed to investigate concomitants of ^.ndividual 
differences in learning. Figure 12.0 outlines the competencies re- 
quired of personnel in the various system-wide evaluation offices. 



Figure 12.0. Optimal Competenaies For- System-Wide Evaluation 
Personnel* 



Senior Staff - 

Institutional 

Research 



Assistants - 

Institutional 

Research 

Senior Staff - 

Long-Range 

Planning 



Assistants - 

Long-Range 

Planning 



1, 2, 3, 6, 7, 10, 16, 19, 23, 24, 26, 27, 30, 

32, 33, 35, 36, 45, 46, 49, 50, 51, 52, 53, 54, 
55, 56, 57, 58, 61, 63, 64, 65, 67, 68, 70, 76, 

77, 78, 80, 82, 83, 84, 85, 87, 88, 90, 91, 
95, 98, 100, 102, 103, 104, 105, 106, 107, 
108, 109, 110, 113, 116, 117, 126, 127, 134, 
135, 136, 137, 138, 141, 145, 146, 152, 155, 

156, 158, 159, 160, 161, 162, 165, 166. 

2, 3, 5, 6, 7, 10, 14, 16, 19, 24, 41, 42, 
4Z, 44, 45, 46, 55, 56, 57, 63, 65, 67, 74, 
75, 76, 77, 85, 91, 98, 100, 103, 104, 105, 
106, 107, 109, 113, 116, 132, 134, 165. 

1, 2, 3, 6, 7, 10, 16, 19, 24, 26, 30, 32, 

33, 35, 36, 38, 55, 56, 57, 63, 64, 76, 77, 

78, 79, 80, 82, 83, 84, 85, 89, 90, 91, 

92, 93, 96, 97, 98, 99, 100, 102, 103, 104, 

106, 107, 108, 109, 110, 113, 116, 123, 124, 

125, 126, 127, 128, 135, 136, 137, 138, 139, 
140, 141, 145, 148, 149, 150, 151, 152, 153, 

157, 160, 161, 162, 165, 166. 

2, 3, 5, 6, 7, 10, 16, 19, 24, 55, 56, 57, 63, 
65, 67, 74, 75, 76, 77, 79, 85, 91, 98, 99, 
100, 103, 104, 106, 107, 116, 123, 124, 125, 

126, 134, 165. 



ERIC 
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Senior staff - h 2, 3, 6, 7, 10, 16, 19, 24, 26, 27, 31, 

Applied Reseopoh 32, 33, 35., 36, 45, 46, 47, 48, 49, 50, 57, 

52, 53, 54, 55, 56, 57, 59, 60, 63, 64, 65, 

67, 70, 71, 76, 77, 80, 85, 91, 98, 100, Wl, 

102, 103, 104, 105, 106, W7, 109, 110, III, 

112, 113, 116, 117, 118, 119, 120, 121, 122, 

131, 132, 134, 135, 136, 137, 138, 141, 144, 

145, 146, 147, 156, 158, 160, 161, 162, 165, 
166. 

Assistants - 2, 3, 5, 6, 7, 10, 14, 16, 19, 24, 41, 42, 

Applied Research 43, 44, 45, 46, 55, 56, 57, 63, 65, 67, 74, 

75, 76, 77, 85, 98, 100, 101, 103, 104, 105, 

106, 107, 113, 116, 117, 132, 164, 165. 

Supvort - 2, 3, 4, 7, 8, 9, 12, 13, 14, 15, 16, 

System-Wide 19, 56, 74, 

Evaluation 

Managemsnt: In situations where individuals are charged 

with the management of subunits, competencies 
163, 164, and 170 also apply. 

'•The competencies listed are those required to function optimally 
in the positions speci-fied. 



Developmental Project Evaluation 

The Developmental Project Evaluation Branch has major responsi- 
bility for most of the input, process, and product evaluation that 
is implement^r. It operates through two offices. Special Project 
Evaluation and Operations Planning. 

The Special Project Evaluation office is responsible for the 
evaluation of those special projects which receive highest priority 
for evaluation from District decision makers. Major activities of 
this office include the formulation and implementation of detailed 
evaluation designs, the design of specialized instruments (in con- 
junction with the Office of Instrument Development and Validation) , 
process evaluation, interim reporting of process and product evalua- 
tion results to project managers, data analysis, and the preparation 
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and reporting of final evaluation reports (see Figure 5,0), In 
addition, this office provides support for program development 

(see Figure 4.0). 

Formal evaluation designs are completed for all projects. 
Figure 13.0 outlines the types of questions that are asked to 
determine whether or not a given evaluation design is adequate. 

Figure 13.0. Criteria For Judging Evaluation Designs 

13.1 Are the objectives of the program adequately 
stated? 

13.2 Are the decision situations to be served ade-- 
quately defined? 

13.3 Are the exmluation questions of interest 
adequately delineated and do they adhere to 
the decision situations to be served? 

13.4 Are the data to be collected adequately spe- 
cified and do they match the evaluation questions 
of interest? Are all questions adequately inves- 
tigated? 

13. b Are the relevant populations and sampling pro- 
cedures for data collection adequately descrzbed? 
Is there reason to believe that the experimental 
and control groups are comparable? 

13.6 Are the instruments for data collecticn adequately 
described? Are they related to the objectives of 
the program? Are they valid and reliable for 

the population being studied? 

13.7 Are schedules sped fied for information collec- 
tion? Are they realistic? 

13.8 Are formats and means for coding ^ organizing^ 
storing i and retrieving data specified? 

13.9 Are means of observing and documenting unantici- 
pated outcomes available? 

13.10 Are data analysis procedures specified,? Are the 
analysis procedures specified appropriate for 
providing useful infomation relative to achieve- 
ment of the objectives of the program and the 
research questions of interest? 
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13.11 Is the evaluation schedule present andy given 
staff and vesauvoe availability^ ift it rea-- 
lis tic? 

13.12 Is the evaluation design likely to provide 
useful (i.e. J valid^ reliable^ objective) 
information? 

IS. IS Are there provisions made for process evalua^ 
tion (i.e.y for observing the project in 
operation) to determine whether or not it is 
functioning according to specifications? 

IS. 14 Are there provisions made for adequate interim 
evaluation? 

IS. IS Is a schedule specified for reporting relevant 
information to specified decision makers? Is 
the schedule realistic? 

IS. 16 Is the budget adequate to carry out the pro- 
posed evaluation? 



Designs are generally longitudinal and include both longitudinal 
and cross-sectional comparison groups, 

xhe Special Prolect Evaluation office is the largest of the 
offices. It includes the evaluation of all federally funded 
projects, as well as all special projects (e.g., special educa- 
tion, bilingual education, etc.), and is composed of many subgroups 
that account for approximately thirty-five percent of the budget. 
Since the Department of Research, Evaluation, and Information Systems 
fulfills an accountability function, care must be taken to ensure 
objective evaluation. Therefore, all project evaluation personnel, 
regardless of their assignments, are members of the central research 
and evaluation staff. Their continued employment is not contingent 
upon continued funding of the particular project or projects to 
which they are assigned. Experience suggests that this arrangement 
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leads to increased objectivity on the part of evaluators and in- 
corporates many of the benefits thought to accrue from both 
external and internal evaluation. Figure 14,0 outlines the 
District's guidelines for project evaluation. 



Figure 14.0. Guidelines For Project Evaluation 

Recognizing the need to ensure the credibility of 
evaluation reports^ all project evaluation staff assigned 
to projects which the Department of Research^ Evaluation, 
and Infomation Systems has the responsibility for eval- ^ 
uating will be directly responsible to the executive officer 
of the Department of Research, Evaluation and Information 
Systems. In addition to ensuring the oredibility of 
evaluation reports, central control of evaluation per- 
sonnel will allow for the most efficient use of human 
resources. Thus, evaluation personnel, regardless of 
location, will be staff members of the Research, Evalua- 
tion, and Information Systems Department and as such, 
subject to the following conditions: 

14.1 Dissemination and Utilization of Evaluation Infomation 

14.1.1 Availability of evaluation information, 
prior to its formal presentation to the 
Board of Education, will be limited to 
those individuals wtio have an estab- 
lished need to know. Normally, this 
would include project officials, as well 
as the evaluation personnel associated 
with the project. 

14.1.2 The proprietary nature of evaluation 
information must be respected. The 
proper functions of evaluation are to 
provide continuous feedback to project 
officials and to furnish relevant infor- 
mation to District decision makers. 

14.1.S All individuals who have access to eval- 
uation information must be cognizant of 
its proper functions and of ethical con- 
siderations applying to the misuse of 
such information. Evaluation personnel, 
particularly, are expected to process 
information in accord with responsible 
reporting procedures. 




14.2 Scope of Work 

14.2.1 What to evaluate^ when to evaluate^ and 
how to evaluate will he determined by 
personnel from Research and Evaluation 
in consultation with those charged with 
the responsibility for making decisions 
about the operation of the project. 

14.2.2 Assignment of duties for evaluation per- 
sonnel shall be made by the executive 
officer of the Department of Research^ 
Evaluation^ and Information Systems or his 
designee. 

14.2.3 All evaluation reports will be delivered 
simultaneously to the project manager and 
the executive officer of the department of 
Research^ Evaluation^ and Information 
Systems. The executive officer of the De- 
partment of Research^ Evaluation^ and 
Information Systems will then forward 
evaluation reports to the executive officer 
of the Development Division^ for pre- 
sentation to the Educational 'Program 

and Evaluation Committee of the Board 
of Education. 

14.3 EmploymtAt 

14.3.1 All evaluation personnel will be recom- 
mended for employment^ promotion^ change 
of project^ or temination of employment 
by the executive officer of the Department 
of Research^ Evaluation^ and Information 
Systems after consultation with the 
Project Director. 

14.3.2 The executive officer of the Department of 
Research^ Evaluation ^ and Information 
Systems will recormend placement of all 
personnel. 

14.3.3 Terms of employment will be the same as 
comparable central office positions. The 
employment security of evaluation per-- 
sonnel assigned to projects will not be 
dependent upon the continued implementation 
of specific projects to which they may be 
assigned. 

14.3.4 All leaves of absence and vacations will be 
subject to approval by the executive officer 
of the Department of Research^ Evaluation^ and 
Information Systems. 
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14.4 Local Regulations and Negotiations 

14.4.1 All evalmtors placed in locations other 
than the Central Administration Building 
shall he subject to the same rules and 
regulations as other administrators in 
that location^ vyith the exception of 
those items specified under Sections 14.2 
and 14.3 above. 

14.4.2 All disagreements relative to the functions 
of project evaluation personnel Dill be 
negotiated by the executive officer of the 
Department of Research^ Evaluation^ and 
Information Systems and personnel responsi- 
ble for making decisions relative to the 
project. 

The Operations Planning Office is responsible for providing technical 
assistance to project managers in planning their projects and to top 
decision makers in utilizing data to make important decisions. In addi- 
tion, this office provides the input evaluation function. Operations 
Planning is currently developing several planning models designed to help 
focus available data at the local school level to aid principals in plan- 
ning their programs. Figure 15.0 outlines the optimal competencies for 
personnel in each of the two offices of Developmental Project Evaluation. 



Figure 15.0. Optimal Corrpctctcics For Developmental Fropcct 
Evaluation Personnel* 

Senior Staff • U 2, 3, 6, 7, 10, 16, 19, 22, 23, 24, 25, 
Special Project 26, 27, 28, 29, 32, 33, 34, 35, 36, 38, 45, 
Evaluation 46, 49, 50, 51, 52, 53, 54, 55, 56, 57, 59, 

60, 63, 64, 65, 67, 70, 76, 77, 80, 85, 91,98, 
100, 101, 102, 103, 104, 105, 106, 107, 109, 
no. 111, 112, 113, 114, 115, 116, 117, 126, 
127, 131, 132, 134, 135, 136, 137, 138, 141, 
142, 143, 145, 146, 152, 156, 158, 160, 161, 
162, 165, 166. 

*The competencies listed are those required to function optimally 
in the positions specified. 
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Assistants - 
Special Pro j eat 
Evaluation 



Senior Staff - 

Operations 

Planning 



Assistants - 

Operations 

Planning 

Support - 
Developmental 
Project 
Evaluation 

Management : 



2, 3, 5, 6, 7, 10, 1 
28, 41, 42, 43, 44, 
65, 67, 74, 75, 76, 
103, 104, 105, 106, 
124, 143, 145, 146, 

1, 2, 3, 6, 16, 19, 
33, 35, 36, 38, 45, 
63, 64, 65, 67, 91, 
103, 104, 106, 107, 
116, 123, 124, 125, 
131, 132, 134, 135, 
160, 161, 162, 165, 

2, 3, 6, 16, 19, 22, 
55, 56, 57, 63, 65, 
99, 100, 103, 104, 1 
123, 125, 128, 129, 

2, 3, 4, 5, 6, 7, 8, 
19, 56, 74, 75. 



6, 19, 22, 23, 24, 25, 

45, 46, 55, 56, 57, 63, 
77, 85, 98, WO, 101, 

107, 109, 113, 114, 116, 
165. 

24, 25, 26, 27. 28, 32, 

46, 47, 48, 55, 56, 57, 
98, 99, 100, 101, 102, 

108, 109, 110, 113, 114, 
126, 127, 128, 129, 130, 
138, 139, 140, 141, 151, 
166. 

24, 25, 38, 45, 47, 48, 
67, 74, 75, 76, 77, 98, 
06, 107, 109, lis, 116, 
130, 134, 165. 

9, 12, 13, 14, 15, 16, 



In situations where individuals are charged 
with the management of subunits, competencies 
163, 164, and 170 also apply. 



Administrative Research 

The Administrative Research Branch has major responsibility for 
responding to the ad hoc needs of District decision makers and for 
providing computer services to the other branches of the Department. 
It operates through the Ad Hoc Research and Computer Applications 
offices. 

No operating research department can survive in an applied set- 
ting without an Office of Ad Hoc Research. If such an office does 
not exist, then the work of the entire department is constantly im- 
pacted by the ad hoc information requirements of decision makers. 
This office is charged with the responsibility of conducting research 
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and evaluation studies to meet Immediate Information demands. In 
addition, personnel from this office provide technical assistance 
In evaluation and research design and Interpretation to District 
personnel, and write the technical sections of many District pro- 
posals. 

The Computer Applications office maintains a longitudinal 
student, staff, and community data base, develops and maintains a 
library of statistical analysis programs, provides the majority 
of Interface with the District's Data Processing Department, and 
provides technical assistance to practicing evaluators In the 
analysis of their data. The most Important task of this office Is 
the maintenance of a longitudinal data base. Figure 16.0 outlines 
some major categories of Information that are Included In the data 
base. 

Figure 16.0. Major Categories Of Information 

Student 

^ permanent student number 
name 

school number (s) 
sex 

ethnic background 
birthdate 

* absences 

^ dropout and withdrawal information 
^ enrollment in special programs 

* raw scores on all system^^wide tests 
^ vocational interest areas 
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aourse mmbers of courses takeriy grades V^-IB 

* grades in courses taken ^ grades 7-12 

* participation in extra-curricular activities ^ 
grades 7-12 

* head of household 

* highest educational level of head of household 

* vocational field of head of household 

* discipline record (suspensions, etc.) 

* one-year follow-up information (a sample) 

* five-year follow-up infomation (a sample) 
Teacher 

* ■ social security monber 

name 

* school number (s) 
sex 

ethnic background 

* birthdate 
teaching experience 

* degree (s) 

institutionis) granting degree (s) 

rmi) scores on available achievement tests 

* course numbers of courses taught 

* absences 
School 

lowest grade 

* highest grade 

* total enrollment by sex and ethnic background within 
grade 

* instructional expenditures by category 

* socioeconomic status of community 

* age, size, and physical condition of facility 

All other necessary data are collected for specific studies through 
random sampling procedures. Figure 17.0 outlines the optimal com- 
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petencies for personnel in each of the two offices of Administrative 



Research. 



Figure 17.0. Optimal Competencies For Administrative 
Research Personnel"^ 



Senior Staff - 
Ad Hoc Research 



Assistants - 
Ad Hoc Reseccrch 



Senior Staff - 

Computer 

Applications 



2, 2, 3, 6, 7, 20, 26, 29, 24, 26, 27, S2, 33, 
35, 36, 38, 45, 46, 49, 50, 52, 52, 55, 56, 57, 
63, 64, 65, 67, 76, 77, 78, 80, 82^ 83, 84, 85, 
92, 98, 200, 202, 202, 203, 204, 205, 206, 207, 
108, 209, 220, 223, 226, 226, 127, 131, 132, 134, 

135, 136, 137, 138, 141, 145, 146, 152, 155, 
156, 158, 160, 161, 162, 165, 166. 

2, 3, 5, 6, 7, 10, 16, 19, 24, 41, 42, 43, 
44, 45, 46, 55, 56, 57, 63, 65, 67, 74, 75, 
76, 77, 85, 98, 100, 101, 103, 204, 105, 106, 
107, 109, 113, 116, 134, 145, 146, 165. 

1, 2, 3, 5, 6, 7, 10, 16, 17, 18, 19, 24, 33, 
58, 59, 60, 61, 63, 64, 65, 67, 76, 77, 78, 
79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 
90, 91, 92, 93, 94^ 95, 96, 102, 103, 104, 
106, no, 113, 123, 124, 126, 127, 134, 135, 

136, 137, 138, 139, 140, 141, 161, 162, 265, 
166. 

2, 3, 5, 6, 7, 10, 16, 17, 18, 19, 24^ 63, 
65, 67, 74, 75, 76, 77, 78, 80, 82, 83, 84, 
85, 87, 89, 90, 91, 92, 94, 103, 123, 124, 
165. 

2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 

16, 17, 18, 19, 56, 74, 75, 76, 77. 

In situations where individuals a:re charged 
with the management of subunits, competencies 
163, 164, and 170 also apply. 

^The competencies listed are those required to optimally function 
in the positions specified. 



Assistants - 

Computer 

Applications 

Support - 
Adrninis trative 
Research 

Management: 



Required Resources 

Research and evaluation is not inexpensive, bu , when used correctly. 
Is cost-effective. If the information obtained from systematic research 
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and evaluation studies is used prudently by decision makers, it can 
save millions of dollars investment in ineffective programs and 
contribute substantially to improving the education of children. 
Thus, the major resource required for the implementation of a re- 
search and evaluation strategy similar to the one outlined in this 
paper is top-level management who are committed to utilizing objec- 
tive data as a major input to educational decisions. If top-level 
management is not so inclined, any research and evaluation effort 
is destined to be noncost-ef f ective and impotent* 

Beyond committment of top-level management to data-based 
decision -roaking, a competent research ano evaluation staff is essen- 
tial. Staff members must possess considerable expertise in measure- 
ment, mathematical statistics, research and evaluation methodology, 
computer programming, data processing and analysis, and report 
generation; and be willing to fill the difficult role of change 
agent. Useful evaluation requires controlled studies that can only 
be implemented by competent methodologists functioning independently 
in applied settings. 

On a more technical basis, a systeiuatic testing program including 
both standardized and criterion-referenced instruments and sufficient 
computer time and facilities to maintain the requisite data bases 
and to permit the accomplishment, of considerable data analysis are 
essential. Without control of data sources and data processing capa- 
bility, the system would fail to produce the timely, reliable infor- 
mal, ion that all such service departments must produce to survive. 
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The type of service described in this paper can be supplied 
to most large school districts (> 50,000 students) for approximately 
two percent of their operating budget. That two percent p s an 
additional two percent for development would provide education with 
a r*inewai syttem that could begin to solve systematically some of 
schooling' 5 difficult problems. 

The establishment of effective research and evaluation systems 
in the public schools is essential for educational renewal. To 
assume that public school systems should be passive recipients of 
externally developed products, often unaccompanied by sufficient 
empirical evidence of success or accompanied by empirical support 
based on restricted student or teacher populations, is ludicrous. 
Often adjustments must be made in externally developed programs to 
meet local requirements, or, where externally developed programs do 
not exist, strategies must be locally developed for meeting specific 
needs. Evaluation, in the absence of applied and basic research 
data, is not capable of providing the necessary information for 
meeting the needs of diverse student populations. 
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Required Competencies for Planning, 
Research, and Evaluation Personnel 

General 

1. Demonstrates a commitment to the empirical approach to 
problem-solving. Draws conclusions on the basis of 
data rather than opinion. 

2. Can clearly and concisely communicate information in 
written form. 

3. Can clearly and concisely communicate information in 
verbal form. 

Support 

4. Can operate transcribing machine. 

5. Can operate keypunch machine. 

6. Can operate electronic calculator. 

7. Can operate sorter. 

8. Can type and proofread technical reports in a rapid 
and professional manner. 

9. Can set up and maintain a numerical or alphabetical 
filing system. 

10. Can operate a time-sharing terminal. 

11. Can operate a PBX Console. 

12. Can operate copying macnines. 

13. Can demonstrate a knowledge of inventory control methods. 

14. Can utilize Post Office weighing scales. 

15. Can operate a Mag Card II typewriter. 

16. Can read and understand writing style manuals and quality 
control technical reports to ensure that they follow the 
accepted formats. 

17. Can operate a remote job entr^' terminal. 

18. Can operate a digital computer. 
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19. Can interpret and implement written and verbal instruc- 
tions in a manner necessary to expedite assigned tasks. 

20. Can administer a line-item budget in such a manner so 
as to ensure that proper expenditure, of funds from many 
diverse funding sources i ^ *»xpedited and that all de- 
partmental bills on received merchandise and maintenance 
are paid. 

21. Can re ■ understand » complete » and, when necessary, 
interac with other departments relative to the myriad 
of special procedures plus supply and personnel forms 
required to function in a large bureaucracy. 

Communications 

22. Can i *'c- with classroom teachers and principals to 
explain the purpose and importance of process evalua- 
tion in providing useful "information to decision makers. 

23. Can interpret standardized test results to professional 
educators ( parents, and students. 

24. Can articulate the purpose of planning, research, and 
evaluation functions in an applied setting. This would 
require the ability to articulate to non-research and 
evaluation personnel the purposes of and interrelation- 
ships among context, input, process and product evalua- 
tion, applied and basic research. 

25* Can provide assistance to program management in estab- 
lishing program objectives and implementation plans. 

26. Can anticipate decisions to be served by a planning, 
research, or evaluation study and design that study to 
provide information relevant to those decisions. 

27. Can articulate the problems of the building principal 
and classroom teacher and provide sufficient lead time 
to allow for necessary intervention in the school 
environment . 

28. Can interact with project management to ensure optimal 
usefulness of project evaluation information. 

29. Can communicate the results of project evaluation to 
project management (this includes differential media 
utilization) . 

30. Can communicate the results of program evaluation to 
District management (this includes differential media 
utilization). 
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31. Can communicate the results of basic research to 
District management (this includes differential 
media utilization)* 

32. Can write both technical and popularized versions of 
planning, research, or evaluation reports, without 
oversimplifying the popularized version. 

33. Can effectively exercise judgment in providing eval- 
uation information to various levels of District 
employees and the public. 

34. Can effectively communicate with District auditors 
and other non-District professional personnel rela- 
tive to design, measurement, data processing, and 
reporting considerations. 

35. Can communicate findings from planning, research, and 
evaluation studies in such a manner that professional 
educators and policy-makers can use the information 
in making and implementing decisions* 

36. Can summarize and interpret planning, research, and 
evaluation literature from other agencies and organi- 
zations in such a manner as is useful to District 
decision makers in planning the future course of edu- 
cation for District students. 

37. Can communicate effectively with policy groups such 
as teacher associations and boards of education con- 
cerning the need for and nature of planning, research, 
and evaluation systems. 

38. Can work effectively in planning tasks with all levels 
of management. 

39. Can provide valid inputs to decision makers at all 
levels in setting priorities for the collection and 
reporting of evaluation information. 

40. Can communicate tlie effects of District projects and 
progr^^ to the mass media and to the public. 

Instrument Construc ti on. Selection, and Implementation 

41. Given a set of specifications, can write forced-choice 
and free-choice test items. 

42. Given a set of specifications, can develop instruments 
to mrasure specific behaviors. These behaviors may be 
in the affective or cognitive domain. 
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43. Given a set of specifications, can design a classroom 
observation instrument. 

44. Given a set of specifications, can design an inter- 
view instrument. 

45. Given an instrument, can conduct an interview. 

46. Can use reference sources such as Buros' Mental Measure- 
ment Yearbook to aid in he selection of standardized 
instruments. 

47. Can write operational objectives at each level of the 
cognitive taxonomy and specify attainment criteria 
associated with them. 

48. Can write operational objectives at each level of the 
affective taxonomy and specify attainment criteria 
associated with them. 

49. Can develop specifications for instruments designed 
to measure specific behaviors. These behaviors can 
be in the cognitive or affective domain. 

50. Can develop specifications for the design of a class- 
room observation instruments 

51. Can develop specifications for the design of an inter- 
view instrument. 

52. Can develop specifications for and implement the 
training of interviewers. 

53. Can design an! implement a classroom observation 
schedule for purposes of process evaluation. 

54. Can develop specifications for and implement the train- 
ing of process evaluators. 

55. Can establish data collection schedules dealing with 
tests, classroom observation, questionnaires, and 
interviews. 

56. Can implement data collection schedules dealing with 
tests, classroom observation, questionnaires, and 
interviews. 

57. Can monitor data collection schedules dealing with 
tests, classroom observation, questionnaires, and 
interviews. 
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58. Can explain the standard procedures that should 
be employed when norining a test. 

59. Can perform an item analysis and interpret the 
results. 

60. Can design and perform an instrument validation 
study and interpret the results. 

61. Can norm a test. 

62. Can establish and implement a system-wide testing 
program incorporating management concepts that per- 
mit continuous monitoring and updating of that 
program. 

easurement 

63. Can read and understand relevant portions of basic 
measurement texts. Such texts are on the level of 
Cronbach (1970), Helmstadtrr (1964), Nunnally (1959), 
and Bracht, Hopkins, & Stanley (1972). (This implies 
an understanding of the topics included in such texts.) 

64. Can read and understand relevant portions of inter- 
mediate measurement texts. Such texts are on the 
level of Gulliksen (1950), Horst (1966), Lindquist 
(1951), Torgerson (1958), and Whitla (1968). (This 
implies an understanding of the topics included in 
such texts.) 

65. Can demonstrate a general understanding of the con- 
cept of reliability. 

66. Can compare and constrast the different method? of 
estimating instrument reliability. 

67. Can demonstrate an understanding of the concept of 
validity and distinguish among content, concurrent, 
predictive, and construct validity. 

68. Can discuss the assumptions underlying the interpre- 
tation of norm-referenced tests. 

69. Can describe the use and limitations of standardized 
tests for placement, diagnostic, and evaluative pur- 
poses, and relate them to crucial issues in testing. 

70. Can compare and constrast the objectives and proce- 
dures of criterion-referenced versus norm-referenced 
measurement . 
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71. Can discuss the assumptions underlying major scaling 
techniques, e.g., subjective estimate, fractionation, 
equisection, differential sensitivity. 

72. Can compare and contrast the different approaches 
to estimating the different types of instrument 
validity. 

73. Can discuss critically emergent developments in the 
field of educational measurement. 

Computer Skills 

74. Can prepare data for analysis either via cardpunch or 
remote terminal given format specifications and instruc- 
tions. 

75. Can code data for preparation given instructions and 
appropriate coding scheme. 

76. Can devise a coding scheme for a given set of data and 
arrange the data and data collection process in such a 
way as to a.ake implementation of the coding scheme 
practical. 

77. Can run existing programs, including preparation of 
all control cards, from either batch processing or 
time-sharing modes given proper instructions. 

78. Writes fluently in a standard applications language 
(e.g., COBOL, FORTRAN). 

79. Is familiar with the nature and function of operating 
systems; understands the relationship between operating 
systems and applications programs. 

80. Understands the characteristics of common storage media 
(e.g., punched cards, magnetic tape, disk). Can analyze 
a data storage application in terms of the advantages 
and limitations of the various media. 

81. Is conversant with the technical vocabulary of data pro- 
cessing. Undt -tands, for example, such teru.3 as register 
stock, flip-flops, masking, interrupt. 

82. Can write programs in ALGOL and/or FORTRAN to perform basi 
statistical calculations on relativ ly small sets of data. 
Statistical calculations include finding means, standard 
deviations, variances, medians, modes, and zero-order 
correlation matrices. 

83. Can write efficient programs for processing large data 
files given appropriate specifications. 
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84 Has some understanding of basic terms and concepts, in- 
cluding such constructs as logical record, random-access, 
microsecond, hexadecimal. 

85. Can accurately state requirements for a data processing 
problem. This involves the ability to specify without 
ambiguity the input and desired output. 

86. Has sufficient overall knowledge to make intelligent recom- 
mendations concerning equipment and configuration. 

87. Given specifications can write programs in ALGOL 
and/or FORTRAN to perform common parametric and non- 
parametric statistical tests including t-test, 
analysis of variance, analysis of covariance, multi- 
ple regression analysis, chi-square contingency 
analysis, and goodness-of-f it tests. 

88. Can provide specifications for the writing of pro- 
grams in ALGOL and/or FORTRAN to perform common 
parametric and nonparametric statistical tests 
including t-test, analysis of variance, analysis 
of covariance, multiple regression analysis, chi- 
square contingency analysis, and goodness-of-f it 
tests. 

89. Can make necessary modifications in existing programs 

to adapt them for use in given problem-solving situa- 
tions. This would include adopting programs from 
other languages and computer installations. 

90. Is thoroughly familiar with internal word formats used 
to represent data. Knows, for example, how the number 
"123" would be represented if treated (a) as a character 
string, (b) as an integer, (c) as a floating number. 

91. Demonstrates an understanding of tne practical limita- 
tions of computer usage. In particular, for a given 
situation can accurately estimate turnaround times, 
programming resources, data preparation resources, 
data coding problems and other practical aspects of 
electronic data processing. 

92. Given specifications, can write programs in ALGOL 
and/or FORTRAN to perform sophisticated projection 
applications. Such applications include first- and 
second-order polynomial regression, Markov pro- 
cesses, and mathematical modeling. 
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93. Can provide specifications for the writing of 
programs in ALGOL and/or FORTRAN to perform sophis- 
ticated projection applications including first and 
second-order polynomial regression, Markov pro- 
cesses, and mathematical modeling. 

94. Given specifications can write programs in ALGOL 
and/or FORTRAN to perform common multivariate app- 
lications. 

95. Can provide specifications for the writing of programs 
in ALGOL and/or FORTRAN to perform common multivariate 
applications. 

96. Demonstrates a knowledge of computer simulation tech- 
niques and major simulation languages* 

97. Cau design and implement a computer simulation of a 
complex educational system. 



Research « Evaluation, and Planning 

98. Can design charts - such as histograms, trend line 
graphs, cross 'reak tables, and p^e graphs - to 
communicate research and/or evaluation findings to 
professional educators and the lay public, 

99. Can develop forms for graphic presentation of planning 
elements and responsibilities (planning, programming, 
budgeting and evaluation matrices). 

100. Can summarize data taken from non-computer based in- 
formation systems. 

101. Can synthesize and provide written reports of the 
results of process evaluation. 
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Can provide concise summaries of technical articles 
published in relevant professional journals (e.g., 
Psvchometrika , Biomtrica ) . 

Can read and understand relevant portions of basic 
design and statistics texts. Such texts are on the 
level of Edwards (1960), Cox (1958), Hays (1963), 
Lindquist (1953), Glass and Stanley (1970), and 
Winer (1962). (This implies an understanding of 
the topics included in such texts •) 
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104. Can understand evaluation and research reports and 
journal articles, which incorporate descriptive 
statistics, e,g., mean, median, standard deviation, 
percentiles, grade-equivalents, correlations. 

105. Can design, implement, write, and/or interpret 
research and/or evaluation reports which incorporate 
descriptive statistics, e.g., mean, median, standard 
deviation, percentiles, grade-equivalents, correlations. 

106. Can understand research and evaluation reports incor- 
porating simple inferential statistics, e.g., t-test, 
simple analysis of variance, point and interval esti- 
mation, as well as survey research and needs assess- 
ment. 

107. Can define a relevant population and formulate sampling 
specifications for data collection purposes utilizing 

a sampling plan incorporating simple random, stratified, 
systematic, or cluster sampling techniques. 

108. Can design, implement, write, and/or interpret a 
needs assessment. 

109. Can design, implement, write, and/or interpret survey 
research using a questionnaire, 

110. Can describe basic concepts involved in parameter esti- 
mation, e.g., bias, precision, consistency, etc., as 
well as the logic of hypothesis testing and inferential 
statistics. 

111. Can describe basic theoretical concepts relating to 
distribution-free statistics, e.g., robustness, 
asymptotic relative efficiency, stochastic inequality, 
location, and scale. 

112. Can apply appropriately the major nonparametric 
techniques, e.g., Mann-Whitney, Wilcoxen, Chi-Square, 
Kruskal-Wallis, Friedman. 

113. Can understand evaluation and research reports and 
journal articles which incorporate univariate analy- 
sis of variance and covariance, discriminant analysis, 
factor analysis, regression analysis, multivariate 
analysis of variance and covariance, Bayesian infer- 
ence and path analysis. 
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114. Understands the data-based approach to curriculum 
construction. 



115. Can design, Implement, and maintain formative 
evaluation systems to aid program managers In 
curriculum construction. 

116. Can review and synthesize research literature In 
a given substantive area. 

117. Understands major educational and psychological 
theories used to represent knowledge of behavior 

and learning processes (e.g.. Skinner, Gagne, Gibson, 
Plaget, Ausubel). 

118. Can relate psychological theories having different 
emphases In a complementary manner (e.g.. Skinner, 
Plaget, Bandura, Chomsky). 

119. Can demonstrate a knowledge of contemporary research 
literature Including areas of Investigation, substan- 
tive findings, and research methods (e.g., Bandura, 
Bower, Flavell, Staats). 

120. Can conceptualize, plan. Implement, and report 
programmatic research designed to systematically 
advance knowledge about the learning process through 
Incorporating established research literature and 
theory Into a longitudinal basic and applied re- 
search program. 

121. Can frame school learning problems as extensions of 
learning and developmental theory and research out- 
comes (e.g.. Bloom, Scandura, Staats, Becker). 

122. Is familiar with the methodology and results of 
aptitude- treatment and trait-trait interaction 
studies (e.g. , Bracht, Cronbach and Snow, Johnson 
and Neyman) . 

123. Can read and understand relevant portions of basic 
operations research texts* Such texts are on the 
level of Sasieni, Yaspan, and Friedman (1959), Beer 
(1966), and Miller and Star (1969). (This Implies 
an understanding of the topics included in such 
texts.) 
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124. Can read and understand relevant portions of basic 
computer simulation texts* Such texts are on the 
level of Mize and Cox (1968) and Hadley (1963). (This 
implies an understanding of the topics included in 
such texts.) 

125. Can demonstrate a knowledge of major planning systems 
and components including Educational Resources Manage- 
ment Systems (ERMS) , Program Evaluation and Review 
Technique (PERT), Management Information Systems (MIS) > 
Programming, Planning, and Budgeting System (PPBS), 
and general systems analysis procedures. 

126. Can understand research and evaluation reports and 
journal articles Incorporating cost-benefit and cost- 
effectiveness analysis, linear programming techniques, 
math modeling, and projections utilizing first and 
second-order polynomial regression and stochastic 
processes. 

127. Can publish in professional journals in specific 
areas of interest. 

128. Can design, implement, write, and/or interpret resource 
allocation reports. 

129. Can reduce a resource allocation to a cost-element 
structure. 

130. Can demonstrate a knowledge of the requisite components 
to be included in a management plan. 

131. Can prepare grant proposals ror funding. 

132. Can articulate current crucial Issues in education 

and where to find research literature relative to them. 

133. Can discuss emergent developments in the field of 
educational evaluation and, in light of those develop- 
ments, distinguish among various conceptualizations 

of evaluation, accountability, research, audit, and 
measurement. 

134. Can outline the salient characteristics of at least 
two major evaluation models and compare and contrast 
the focal points of each. 
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135. Can describe the application of and the relation- 
ship existing among the major theoretical distri- 
butions of statistics, e,g., the normal, 
multivariate normal, binomial, hypergeometric , 
poisson, chi-square, and variance- ratio. 

136. Can describe basic concepts of analysis of variance, 
e.g., design matrix, random effect, fixed effect, 
interaction, confounding, and error and relate those 
concepts to various ANOVA designs. 

137. Can relate traditional analysis of variance to re-~ 
gression analysis (the general linear model) and 
describe the effects of utilizing various least 
squares estimation approaches (e.g., backword 
elimination, forward selection, stepwise regression) 
on observed results. 

138. Can read and understand relevant portions of inter- 
mediate design and statistics texts. Such texts are 
on the level of Draper and Smith (1966), Morrison 
(1967), and Whitla (1968). (This implies an under- 
standing of the topics included in such texts.) 

139. Can read and understand relevant portions of inter- 
mediate operations research texts. Such texts are 

on the level of Churchman, Ackoff, and Arnoff (1957), 

Hillier and Lieberraan (1967), and Wagner (1969). 

140. Can read and understand relevant portions of inter- 
mediate computer simulation texts. Such texts are 
on the level of Chorafas (1965) and Emshoff and 
Sisson (1970). 

141. Can discuss the logic of statistical analysis, the 
major classes of questions that can be addressed by 
present modes of analysis, the classes of assumptions 
that can be accommodated by present statistical tech- 
nology, and the emergent developments in the field. 

142. Can present a case for the evaluation of competing 
instructional strategies and suggest alternate 
methodologies which might be used. 

143. Can design and monitor evaluation studies that monitor 
the implementation of projects. 
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144. Can design, implement, write, and/or interpret con- 
trolled research studies that are focused on 
improving the educational process. 

145. Can develop and implement an evaluation or research 
design to fit a particular applied situation that 
evidences adequate sampling procedures, instrumenta- 
tion, planned information collection, data analysis, 
and controls for threats to the internal and external 
validity of evaluation results (this encompasses many 
other competencies and can be done at the level of 
the individual, e.g., at competency level 146 or at 
competency level 158) . 

146. Can design, implement, write, and/or interpret re- 
search or evaluation reports incorporating simple 
inferential statistics, e.g., t-test, simple analy- 
sis of variance, point and interval estimation. 

147. Can design, implement, wrilte, and/or interpret 
aptitude-tr3atment and trait-trait interaction 
studies. 

148. Can design, implement, write, and /or interpret pro- 
jection reports utilizing first and second-order 
polynomial regression. 

149. Can design, implement, write, and/or interpret pro- 
jection reports utilizing stochastic processes. 

150. Can synthesize a mathematical model of a complex 
system from written and verbal input of non- 
technical managers. 

151. Can operationalize major planning systems in an 
educational setting including Educational Resources 
Management Systems (ERMS) , Program Evaluation and 
Review Technique (PERT), Management Information 
Systems (MIS), Programming, Planning and Budgeting 
System (PPBS) , and general systems analysis procedures. 

152. Can design, implement, write, and/or interpret reports 
incorporating cost-benefit and cost-effectiveness 
analysis. 

153. Can design, implement, write, and/or interpret reports 
utilizing the principles of operations research. 

154. Can specify, operationalize, and apply criteria for 
evaluating planning and evaluation systems. 
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156. Can design, Implement, write, and/or Interpret 
research and/or evaluation reports Incorporating 
higher order analysis of variance and covarlance, 
discriminant analysts, and regression analysis. 

157. Can design. Implement, write, and/or Interpret 
reports Incorporating linear programming tech- 
niques. 

158. Can design. Implement, write, and/or Interpret 
research and/or evaluation reports Incorporating 
factor analysis and multivariate analysis of 
variance and covarlance. 

159. Can conceptualize a set of performance Indicators 
that would form a sound basis theoretically and 
practically for the School District's systematic 
evaluative Information system. 

160. Can critique an evaluation or research design In 
terms of Its Internal and external validity, 
appropriateness of methodology, appropriateness 
of Instrumentation, etc. 

Management 

161. Can provide realistic resource estimates for a 
specific planning, research* or evaluation project. 

162. Can systematically plan the Implementation of a 
planning, research, or evaluation project. Con- 
siderations Include timeline, required resources, 
and specification of factors that might Impinge 
on successful completion. 

163. Can effectively control, direct, and supervise 
subordinates. This Includes the utilization of 
employee work time In a most expedient and bene- 
ficial manner, the establishment and Implementa- 
tion of a fair and objective employee evaluation 
system, the periodic review of Individual progress 
toward assigned goals, and the handling of personal 
personnel problems In a discrete and emphathetlc 
manner. 

164. Can effectively articulate and Interpret School 
District and departmental policy to subordinates. 
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165. Can maintain good rapport with other District 
departments and represent the department in a 
cooperative and productive manner at numerous 
formal and informal meetings with District 
personnel. 

166. Can maintain good rapport and represent the Dis- 
trict in a cooperative and productive manner at 
numerous forn.<.l and informal meetings with personnel 
from other agencies engaged in planning, research, 
and evaluation activities. 

167. Can plan, procure, and maintain a departmental 
library. 

168. Can supervise custodial maintenance of the physical 
plant. 

169. Can provide realistic resource estimates for a plrm- 
ntng, research, and evaluation program and allocate 
those resources to accomplish stated goals. This 
includes the preparation and administration of a 
program budget. 

170 Can establish and implement a management-by-objectives 
system for a planning, research, and evaluation unit. 
This includes a management control system that monitors 
the implementation of all research and evaluation 
projects undertaken by the unit. 

171 Can establish and implement, within the guidelines set 
forth by affirmative action, a systematic recruiting 
system for planning, research, evaluation, and computer 
science personnel. 

172. In'tiates and obtains required administrative and 
Board approval on all personnel documents and arbitrates 
any personnel disputes. 

173. Can both envision and articulate (in verbal and written 
form) longitudinal and cross-sectional departmental 
goals as they relate to District goals and policy. 

174. Can use evaluation information to effect rational deci- 
sions about departmental goals and strategies. 

175 Can negotiate the political system or the parent 

institution in stich a manner as to ensure that objec- 
tive data reach principal decision cakers and that 
planning, research, and evaluation activities are 
valued and supported. 
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